Real-time video processing for pyramid holographic projections

ABSTRACT

Various techniques are described herein that provide for a real-time image or video processing system that is able to capture and stream or record/store video content of an object, and turn the captured content into a new video format that can be properly projected onto a pyramid holographic projector. In one specific embodiment, the techniques herein capture a video selfie of a user and stream it live or else store it for playback later as a saved message. Other embodiments, such as controlled avatars, animated characters, etc., may also be converted from a standard 2D format into a pyramid hologram format, either in real-time or else during post-processing, accordingly.

RELATED APPLICATION

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/636,988, filed Mar. 1, 2018, entitled REAL-TIME VIDEO PROCESSINGFOR PYRAMID HOLOGRAPHIC PROJECTIONS, by Bezirganyan et al., the contentsof which are hereby incorporated by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates generally to video processing, and, moreparticularly, to real-time video processing for pyramid holographicprojections.

BACKGROUND

Since around 2010, hollow pyramid-shaped prisms have been placed over aflat screen (or smartphone) by consumers to emulate a three-dimensional(3D) image by means of two-dimensional (2D) light refraction. Forinstance, early stage crafters would use various sources of plastic(e.g., a plastic CD cover) cut and assembled into a pyramid shape toturn their smartphones into hologram projectors. Such hologramprojectors (also known as a “holography pyramid” or “holographydisplay”) make the holographic 3D projection possible based on a conceptcalled the “Pepper's Ghost Illusion”, which is an illusion techniquewhere a picture of an object or person is displayed on a flat surface(also referred to as a “bounce”), which is at an approximate 45-degreeangle to a transparent screen surface. The pyramid hologram projector,therefore, acts as the transparent screen surface in multiple (e.g.,four) directions, and multiple (e.g., four) images are displayed forreflection by the transparent surfaces of the pyramid to be viewed fromrespective directions, accordingly.

Though there are various simplified videos available online today thatcan take advantage of this technology (e.g., butterflies, fireworks,etc.), the processing of these “pyramid hologram” demonstration videoshas been time-consuming and performed offline by skilled graphicartists.

SUMMARY

According to embodiments herein, various techniques provide for areal-time image or video processing system that is able to capture andstream or record/store video content of an object, and turn the capturedcontent into a new video format that can be properly projected onto apyramid holographic projector. In one specific embodiment, thetechniques herein capture a video selfie of a user and stream it live orelse store it for playback later as a saved message. Other embodiments,such as controlled avatars, animated characters, etc., may also beconverted from a standard 2D format into a pyramid hologram format,either in real-time or else during post-processing, accordingly.

Other specific embodiments, extensions, or implementation details arealso described below.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments herein may be better understood by referring to thefollowing description in conjunction with the accompanying drawings inwhich like reference numerals indicate identically or functionallysimilar elements, of which:

FIG. 1 illustrates an example of well-known holographic projectiontechniques;

FIG. 2 illustrates an alternative arrangement for a projection-basedholographic projection system, namely where the projector is located onthe floor, and the bounce is located on the ceiling;

FIG. 3 illustrates an example of a holographic projection system usingvideo panel displays, with the panel below a transparent screen;

FIG. 4 illustrates an example of a holographic projection system usingvideo panel displays, with the panel above a transparent screen;

FIG. 5 illustrates an example simplified holographic projection system(e.g., communication network);

FIG. 6 illustrates a simplified example of an avatar control system;

FIGS. 7A-7B illustrate an example of pyramid holographic projections;

FIG. 8 illustrates an example of an alternative configuration of pyramidholographic projections;

FIG. 9 illustrates an example of another alternative configuration ofpyramid holographic projections;

FIG. 10 illustrates an example of an image configuration for pyramidholographic projections;

FIGS. 11A-11E illustrate an example of real-time video processing forpyramid holographic projections;

FIGS. 12A-12B illustrate examples of a depth-based video capture device;

FIGS. 13A-13D illustrate an example of depth-based video capture;

FIG. 14 illustrates an example of enhanced image processing;

FIG. 15 illustrates an example simplified procedure for depth keycompositing;

FIG. 16 illustrates an example simplified procedure for real-time videoprocessing for pyramid holographic projections;

FIG. 17 illustrates an example of a pyramid hologram selfie environment;

FIG. 18 illustrates an example closed travel case;

FIGS. 19A-19B illustrate an example of an opened travel case for pyramidholographic projections; and

FIG. 20 illustrates an example simplified computing device.

DESCRIPTION OF EXAMPLE EMBODIMENTS

The “Pepper's Ghost Illusion” is an illusion technique known forcenturies (named after John Henry Pepper, who popularized the effect),and has historically been used in theatre, haunted houses, dark rides,and magic tricks. It uses plate glass, Plexiglas, or plastic film andspecial lighting techniques to make objects seem to appear or disappear,become transparent, or to make one object morph into another.Traditionally, for the illusion to work, the viewer must be able to seeinto a main room, but not into a hidden room. The hidden room may bepainted black with only light-colored objects in it. When light is caston the room, only the light objects reflect the light and appear asghostly translucent images superimposed in the visible room.

Notably, Pepper's Ghost Illusion systems have generally remained thesame since the 19th Century, adding little more over time than the useof projection systems that either direct or reflect light beams onto thetransparent angled screen, rather than using live actors in a hiddenroom. That is, technologies have emerged in the field of holographicprojection that essentially mimic the Pepper's Ghost Illusion, usingprojectors as the light source to send a picture of an object or personwith an all-black background onto a flat, high-gain reflection surface(also referred to as a “bounce”), such as white or grey projectionscreen. The bounce is typically maintained at an approximate 45-degreeangle to the transparent screen surface.

FIG. 1 illustrates an example of a conventional (generally large-scale)holographic projection system 100 (e.g., demonstrating a recent trend inlive music performances using a holographic projection of a performer,such as a live-streamed, pre-recorded, or re-constructed performance).Particularly, the streamed (or recorded, or generated) image of theartist (or other object) may be projected onto a reflective surface,such that it appears on an angled screen and the audience sees theartist or object and not the screen. If the screen is transparent, thisallows for other objects, such as other live artists, to stand in thebackground of the screen, and to appear to be standing next to theholographic projection when viewed from the audience.

FIG. 2 illustrates an alternative arrangement for a projection-basedholographic projection system, namely where the projector 210 is locatedon the floor, and the bounce 240 is located on the ceiling. The stickfigure illustrates the viewer 260, that is, from which side one can seethe holographic projection. In this arrangement, the same effect can beachieved as in FIG. 1, though there are various considerations as towhether to use a particular location of the projector 210 as in FIG. 1or FIG. 2.

Though the projection-based system is suitable in many situations,particularly large-scale uses, there are certain issues with usingprojectors in this manner. For example, if atmosphere (e.g., smoke froma fog machine) is released, the viewer 260 can see where the light iscoming from, thus ruining the effect. Also, projectors are not typicallybright enough to shine through atmosphere, which causes the reflectedimage to look dull and ghost-like. Moreover, projectors are large andheavy which leads to increased space requirements and difficultyrigging.

Another example holographic projection system, therefore, with referencegenerally to FIGS. 3 and 4, may be established with video panel displays270, such as LED or LCD panels, mobile phones, tablets, laptops, ormonitors as the light source, rather than a projection-based system. Inparticular, these panel-based systems allow for holographic projectionfor any size setup, such as from personal “mini” displays (e.g., phones,tablets, etc.) up to the larger full-stage-size displays (e.g., withcustom-sized LCD or LED panels). Similar to the typical arrangement, apreferred angle between the image light source and the reflective yettransparent surface (clear screen) is an approximate 45-degree angle,whether the display is placed below the transparent screen (FIG. 3) orabove it (FIG. 4).

Again, the stick figure illustrates the viewer 260, that is, from whichside one can see the holographic projection. Note that the systemtypically provides about 165-degrees of viewing angle. (Also note thatvarious dressings and props can be designed to hide various hardwarecomponents and/or to build an overall scene, but such items are omittedfor clarity.)

The transparent screen is generally a flat surface that has similarlight properties of clear glass (e.g., glass, plastic such as Plexiglasor tensioned plastic film). As shown, a tensioning frame 220 is used tostretch a clear foil into a stable, wrinkle-free (e.g., and vibrationresistant) reflectively transparent surface (that is,displaying/reflecting light images for the holographic projection, butallowing the viewer to see through to the background). Generally, forlarger displays it may be easier to use a tensioned plastic film as thereflection surface because glass or rigid plastic (e.g., Plexiglas) isdifficult to transport and rig safely.

The light source itself can be any suitable video display panel, such asa plasma screen, an LED wall, an LCD screen, a monitor, a TV, a tablet,a mobile phone, etc. A variety of sizes can be used. When an image(e.g., stationary or moving) is shown on the video panel display 270,such as a person or object within an otherwise black (or other stabledark color) background, that image is then reflected onto thetransparent screen (e.g., tensioned foil or otherwise), appearing to theviewer (shown as the stick figure) in a manner according to Pepper'sGhost Illusion. However, different from the original Pepper's GhostIllusions using live actors/objects, and different from projector-basedholographic systems, the use of video panel displays reduces oreliminates the “light beam” effect through atmosphere (e.g., fog),allowing for a clearer and un-tainted visual effect of the holographicprojection. (Note that various diffusion layers may be used to reducevisual effects created by using video panel displays, such as the Moiréeffect.) Also, using a video panel display 270 may help hide projectorapparatus, and may reduce the overall size of the holographic system.

Additionally, some video panels such as LED walls are able to generate amuch brighter image than projectors are able to generate thus allowingthe Pepper's Ghost Illusion to remain effective even in bright lightingconditions (which generally degrade the image quality). The brighterimage generated from an LED wall also allows for objects behind the foilto be more well-lit than they can be when using projection. Moreover, bydisplaying an image of an object or person with a black background onthe light source, it is reflected onto the transparent flat surface soit looks like the object or person is floating or standing on its own.

In general, holographic projections may be used for a variety ofreasons, such as entertainment, demonstration, retail, advertising,visualization, video special effects, and so on. The holographic imagesmay be produced by computers that are local to the projectors or videopanels, or else may be generated remotely and streamed or otherwiseforwarded to local computers.

As an example, by streaming the video image of the performer as a videoand projecting it onto a holographic projection system, a true concertor nightclub experience can be transmitted across the globe for the liveentertainment experience. For instance, holographically live-streamingconcerts to satellite venues around the globe while maintaining the liveconcert experience helps artists reach new markets and new revenuestreams, while bringing live sets to more fans all across the world.Satellite venues can be configured to have the same concert feel as anactual show: intense lighting effects, great sound quality, bars,merchandise, etc. The only difference is that the performers are notphysically present, but are holographically projected from the broadcastvenue. The music may be streamed directly from the soundboard of thebroadcast venue and sent to state-of-the-art sound systems at thesatellite venues, where light shows may accompany the performance withtop of the line LED screens and lasers.

For instance, FIG. 5 illustrates an example simplified holographicprojection system (e.g., communication network), where the network 500comprises one or more source A/V components 510, one or more “broadcast”computing devices 520 (e.g., a local computing device), a communicationnetwork 530 (e.g., the public Internet or other communication medium,such as private networks), one or more “satellite” computing devices 540(e.g., a remote computing device), and one or more remote A/V components550.

In the example above, a broadcast venue may comprise the source A/Vcomponents 510, such as where a performance artist is performing (e.g.,where a disc jockey (DJ) is spinning) in person. The techniques hereinmay then be used to stream (relay, transmit, re-broadcast, etc.) theaudio and video from this broadcast location to a satellite venue, wherethe remote A/V components 550 are located. For instance, the DJ in thebroadcast location may have the associated audio, video, and evencorresponding electronic effects (lights, pyrotechnics, etc.) streameddirectly to the satellite venue's A/V system with the same high qualitysound as if the musician/artist was playing/singing in person.

As another example, in computing, an “avatar” is the graphicalrepresentation of the user (or the user's alter ego or other character).Avatars may generally take either a two-dimensional (2D) form orthree-dimensional (3D) form, and typically have been used as animatedcharacters in computer games or other virtual worlds (e.g., in additionto merely static images representing a user in an Internet forum). Tocontrol an avatar or other computer-animated model (where, notably, theterm “avatar” is used herein to represent humanoid and non-humanoidcomputer-animated objects that may be controlled by a user), a userinput system converts user action into avatar movement.

FIG. 6 illustrates a simplified example of an avatar control system. Inparticular, as shown in the system 600, a video capture/processingdevice 610 is configured to capture video images of one or more objects,particularly including one or more users 620 that may have an associatedposition and/or movement 625. The captured video data may comprise colorinformation, position/location information (e.g., depth information),which can be processed by various body tracking and/or skeletal trackingalgorithms to detect the locations of various tracking points (e.g.,bones, joints, etc.) of the user 620. An avatar mapping system 650 maybe populated with an avatar model 640, such that through various mappingalgorithms, the avatar mapping system is able to animate an avatar 665on a display 660 as controlled by the user 620. Illustratively, inaccordance with the techniques herein the display 660 may comprise aholographic projection of the model animated avatar 665, e.g., allowingan individual to interactively control a holographic projection of acharacter. (Notably, the avatar mapping system 650 may provide itscontrol functionality in real-time or as a recorded/post-productionvideo feed, and may be co-located with the video processing system 630,remotely located from the video processing system, or as dividedcomponents allowing it to be both local to and remote from the videoprocessing system.)

—Pyramid Holographic Projections—

As noted above, pyramid-shaped prisms can be placed over a flat screen(such as a tablet or smartphone) by consumers to emulate a 3D image bymeans of 2D light refraction. Such hologram projectors (also known as a“holography pyramid” or “holography display”) make the holographic 3Dprojection possible based on the “Pepper's Ghost Illusion”.

FIG. 7A illustrates an example pyramid holographic projection system 700that may be used in accordance with one or more embodiments herein. Inparticular, each flat surface of the pyramid hologram projector 710 actsas a transparent screen surface in a respective direction (e.g., four,as shown, though three or more than four may also be used with similarlycorresponding images), and is placed on the surface of an image displaydevice 720 (e.g., smartphone). Images 730 may then be displayed forreflection as holographic projection 740 by each of transparent surfacesof the pyramid, and, as shown in FIG. 7B, to be viewed by users 750 fromrespective directions, accordingly. (Note that while the same image 730may be copied for each of the plurality images shown, other embodimentsmay display different images, such as described below.)

Additionally, as shown in FIG. 8, an alternative system 800 may beembodied as a stand 810 holding the pyramid hologram projector 710 in aflipped orientation, where the image source (display device 720) may besupported from the top of the stand 810, as shown.

It should be specifically noted that while the embodiments shown inFIGS. 7A-8 are based on an illustrative smartphone, any suitable displaydevice, include both panel displays and projection displays, may be usedwith the techniques herein (e.g., depending on the size of the display).Also, while a four-sided pyramid is shown with equal sides, otherarrangements of the pyramid may be used, such as a three-sided pyramid.Note further that not all sides need to be configured to display animage, such as the alternative embodiment 900 further shown in FIG. 9,where a three-sided pyramid (projector) 910 is shown within a stand 920that does not have equally shaped sides (displaying images 730 forreflections/holographic projections 740), such as for displays upagainst a wall (e.g., a back of a display that is not meant for users tobe behind).

Furthermore, FIG. 10 illustrates a corresponding display 1000 for imagesuite 1010, comprising each individual image 730 (e.g., 730 a-730 d, asshown). Note that the reference numbers 730 a-740 d do not specificallypoint to the illustrative female icon, but instead reference the entirecorresponding portion of the image suite 1010, since the entiredisplayed area is the image 730. Typically, however, the icon or otherobject (e.g., a person) would be displayed in front of a blackbackground, thus the only image reflected would be the object itself. Inaddition to the individual images 730, the image suite 1010 may alsocomprise a reference locator 1020, in order to ensure that acorresponding pyramid projector is places in the proper position andorientation for proper operation. As further demonstrated, since theaspect ratio of most displays are not square (e.g., 16:9), the fulldisplay 1000 may comprise one or more blackened areas 1015 to allow forsymmetrical display of the images 730 (e.g., a 9:9 ratio for image suite1010, with the additional portion(s) of the 16:9 aspect screen beingblankly displayed).

As also mentioned above, though there are various simplified videosavailable online today that can take advantage of pyramid hologramtechnology, the processing of these images/videos has beentime-consuming and performed offline by skilled graphic artists. Thetechniques herein, therefore, provide a real-time image or videoprocessing system that is able to capture and stream or record/storevideo content of an object, and turn the captured content into a newvideo format that can be properly projected onto a pyramid holographicprojector. In one specific embodiment, the techniques herein capture avideo selfie of a user and stream it live or else store it for playbacklater as a saved message. Other embodiments, such as controlled avatars,animated characters, etc., may also be converted from a standard 2Dformat into a pyramid hologram format, either in real-time or elseduring post-processing, accordingly.

Operationally, the techniques herein take a video input and process itthrough a custom script (e.g., that runs on an FFMPEG framework) toconvert that video into a pyramid holographic format. As mentioned, thisconversion may be performed real-time or else post recording on aserver.

FFMPEG (generally referring to a “fast forward” project based on theMoving Picture Experts Group (MPEG) international standard for encodingand compressing video images), as a particular illustrative example, isa software project that produces libraries and programs for handlingmultimedia data. The FFMPEG framework is based on a suite of open sourcesoftware that permits managing of audio or video streams to makerecordings, corrections with filters, and/or transcode media from oneformat to another (e.g., decode, encode, transcode, mux, demux, stream,filter, play, etc.).

The script running on the illustrative framework (e.g., FFMPEG orotherwise) can be based on a graphics engine such as the cross-platformUnity engine developed by Unity Technologies, as will be understood bythose skilled in the art. Graphics engines, in particular, may be usedto develop both three-dimensional and two-dimensional models and videosand simulations for computers, consoles, and mobile devices. Unity, forexample, allows importation of sprites and an advanced 2D world rendererfor 2D videos, while for 3D videos, it allows specification of texturecompression, mipmaps, and resolution settings for each platform that thegraphics engine supports, and provides support for bump mapping,reflection mapping, parallax mapping, screen space ambient occlusion(SSAO), dynamic shadows using shadow maps, render-to-texture, andfull-screen post-processing effects. Unity also supports the creation ofcustom vertex, fragment (or pixel), tesselation, compute shaders, andsurface shaders. Notably, graphics engines, such as Unity, also allowfor the creation of scenes, which are a collection of objects typicallyoperated on as a unit during execution of the video (e.g., whereenvironments, obstacles, and decorations, can be defined).

According to the techniques herein, a graphics processing unit (GPU),especially with accelerated compression and compositing, can take asingle image/video source, and can produce a pyramid holographicprojection video source by optionally first copying the same image/videomultiple times, and overlaying the final result on top of a transparentframe, positioning and rotating each image/video to form an “open box”shape that is centered in the frame.

With reference to the example 1100 of FIGS. 11A-11E, the techniquesherein are configured to create a pyramid holographic projection in realtime, using a GPU script such as within the illustrative softwareenvironments described above. For instance, as shown in step 1101 ofFIG. 11A, an initial input image 1110 (e.g., a frame of a video or astill image) is captured (e.g., input from selfie camera, as describedbelow), and then with a flat planar “UV map” (triangle 1120), only whatis in the triangle will be rendered (i.e., not the removed portions1125).

Notably, UV mapping is the 3D modeling process of projecting a 2D image(texture map) to a 3D model's surface for texture mapping (the letters“U” and “V” denote the axes of the 2D texture because “X”, “Y”, and “Z”are already used to denote the axes of the 3D object in model space).That is, UV texturing permits polygons that make up a 3D object to bepainted with color (and other surface attributes) from an ordinaryimage, called a UV texture map. As will be appreciated by those skilledin the art, the UV mapping process involves assigning pixels in theimage to surface mappings on the polygon, usually done by“programmatically” copying a triangular piece of the image map andpasting it onto a triangle on the object.

If only a square or rectangle (e.g., the original image 1110) were to beused, which has a “face UV map”, the GPU would create the kind ofdistortion 1130 as shown in step 110X (not a step of the techniquesherein) in FIG. 11B. However, according to the techniques herein, byusing a trapezoid shape 1120 with a flat planar map, as shown in step1102 of FIG. 11C the ultimate image 1140 (planar map applied to atrapezoid) thus avoids the distortions. (Note that the bottom left andright corners of the shape will remain invisible, i.e., the cut-outportions 1125 from FIG. 11A).

As shown in step 1103 of FIG. 11D, instead of using the rectangularshape for the video “quad”, the techniques herein use the trapezoidshape image 1140 (which has a flat UV map), and copies the image to beprojected on all sides of the pyramid (e.g., four sides) as final image1150. Note that in accordance with one embodiment of the techniquesherein, the original image may be a single image (e.g., from a singlecamera), and as such the image may be copied four times to be equallydisplayed on each of the four surfaces (of a four-surface pyramid).However, in other embodiments, multiple cameras may be used (e.g., fourcameras: front, back, right side, left side), and as such, the copyingof the image 1140 need not occur, however step 1103 would still stitchthe multiple images from the multiple cameras together to create thefinal image 1150, accordingly. (Note further that the center of theimage 1150 may be configured as the reference locator 1020 for theappropriate pyramid template, as mentioned above.)

The final result of step 1104 in FIG. 11E is an image or video stream asa pyramid holographic projection 1160, where the input image/video isconverted and transformed in real time. In other words, the trapezoidcan be used to create the pyramid shape with the same image projected.Other effects, such as customizing the aspect ratio size and additionalcontent can also occur in either real-time or in post processing.

According to one or more specific embodiments of the present disclosure,the techniques above may be used with advanced “holographic selfie”technology, where the user is separated from the environment, and onlythe user (or other object) is recorded or streamed as a video. That is,as described below, pyramid calling or messaging (or other videostreams) may be established by producing a holographic selfie video andconverting it into the pyramid projection image in real-time asdescribed above.

As one example, Chroma Keying or Chroma Key Compositing is generally apost-production technique of layering two film images together based oncolor. For example, as is well understood in the art, a person or objectmay be filmed in front of a “green screen” (though any color may beused), and the green color is replaced through software with anotherbackground image. One problem with such an approach, however, is that itrequires a solid color background, where the person or object must beplaced between a camera and the solid color in order for the ChromaKeying to work properly. Another problem is that the environment must becarefully planned so that the person or object does not have any of thesolid color (e.g., green) on them, such as a shirt or tie, otherwise thesoftware mistakenly detects the color as something to replace, resultingin strange artifacts of a background image appearing on the person orobject.

A similar technique that does not require a solid color background mayremove background objects based on a tracked user being specified bytheir skeletal recognition. In particular, this technique uses variousimage processing techniques to select and track a single person as theforeground, and remove the rest of the background from the scene.Notably, however, this technique currently does not allow for multiplepeople to be set as the foreground, nor does it allow for any non-humanobjects to be considered as the foreground (or a part thereof). Also,this technique requires a stagnant background (e.g., the tracked personshould stand in a relatively uncluttered space, avoid standing in frontof a very dark background or very bright light source pointing towards asensor, and avoid holding a large reflective item), and the person orobject cannot leave the frame.

An advanced technique herein addresses these problems, allowing a personor object can be filmed in any environment, while allowing for theseparation of the person or object from its surrounding background inreal-time, regardless of the background in use, and while allowing themto exit and re-enter the frame. In particular, certain embodimentsherein can be configured to visually capture a person and/or object froma video scene based on depth, and isolate the captured portion of thescene from the background in real-time.

In order to accomplish depth-based keying in this manner, a videocapture device used herein may comprise a camera that is capable ofdetecting object distance. One such example camera that is commerciallyavailable is the KINECT camera, available from MICROSOFT.Illustratively, as shown in FIG. 12A, a depth-based video capture device1200 may comprise two primary components, namely a video camera 1210 anda depth-capturing component 1220. For example, the video camera 1210 maycomprise a “red, green, blue” (RGB) camera (also called a color videographics array (VGA) camera), and may be any suitable rate (e.g., 30 or60 frames per second (fps)) and any suitable resolution (e.g., 640×480or greater, such as “high definition” resolutions, e.g., 1080p, 4K,etc.).

The depth capturing component 1220 may comprise two separate lenses, asillustrated in FIG. 12B, such as an infrared (IR) emitter 1222 to bathethe capture space in IR light, and an IR camera 1224 that receives theIR light from the IR emitter as it is reflected off of the objectswithin the capture space. For instance, the brighter the detected IRlight, the closer the object is to the camera. One specific example ofan IR camera is a monochrome CMOS (complementary metal-oxidesemiconductor) sensor. Notably, the IR camera 1224 (or depth capturingcomponent 1220, generally) may, though need not, have the same framerate and resolution as the video camera 1210 (e.g., 30 fps and 640×480resolution). Note also that while the video camera 1210 and depthcapturing component 1220 are shown as an integrated device, the twocomponents may be separately located (including separately locating theillustrative IR emitter 1222 and IR camera 1224), so long as there issufficient calibration to collaboratively determine portions of thevideo image based on depth between the separately located components.

Based on inputting the images from the camera (e.g., a source A/Vcomponent) into the broadcasting computing device, a corresponding depthkey compositing process enables setting/defining a desired depth range(e.g., manually via a user interface, or dynamically by the processitself) using the captured depth information (e.g., IR information). Forexample, FIG. 13A illustrates an example source image 1310 that may becaptured by the video camera 1210. Conversely, FIG. 13B illustrates anexample depth-based image 1320 that may be captured by the depthcapturing component 1220, such as the IR image captured by the IR camera1224 based on reflected IR light from the IR emitter 1222. Inparticular, the image 1320 in FIG. 13B may be limited (manually ordynamically) to only show the desired depth range of a given subject(person, object, etc.), such as based on the intensity of the IRreflection off the objects.

According to one or more embodiments herein, the depth range selected toproduce the image 1320 in FIG. 13B may be adjusted on-the-fly (e.g.,manually by a technician or dynamically based on object detectiontechnology) in order to control what can be “seen” by the camera. Forinstance, the techniques herein thus enable object tracking during liveevents, such as individual performers move around a stage. For example,as shown in FIG. 13C, an aerial view of the illustrative scene is shown,where the desired depth range 1330 may be set by a “near” depththreshold 1334 and a “far” depth threshold 1332. As an example, a usermay be prompted to press the ‘−’ or ‘+’ keys on a keyboard to decreaseand increase the near threshold, respectively, and the ‘<’ or ‘>’ keysto correspondingly decrease and increase the far threshold,respectively. Other techniques (and particularly user inputs/keys) maybe made available, such as defining a center depth (distance fromcamera) and then a depth of the distance captured surrounding thatcenter depth, or defining a near or far depth threshold and then afurther or nearer depth (in relation to the near or far depththreshold), respectively. This can also be combined with other bodytracking algorithms (e.g., as described below).

By then overlaying the depth information (IR camera information) ofimage 1320 in FIG. 13B with the video image 1310 from FIG. 13A, thetechniques herein “cut out” anything that is not within a desired depthrange, thus allowing the camera to “see” (display) whatever is withinthe set range, as illustrated by the resultant image 1340 in FIG. 13D.In this manner, the background image may be removed, isolating thedesired person/object from the remainder of the visual scene captured bythe video camera 1210. (Note that foreground images may also thus beremoved, such as for various visual effects other than thosespecifically mentioned herein.)

By maintaining a consistent depth range 1330, a mobile object or personmay enter or exit the depth range, thus appearing and disappearing fromview. At the same time, however, by allowing for the dynamic andreal-time adjustment of the depth range as mentioned above, a mobileobject or person may be “tracked” as it moves in order to maintainwithin the depth range, accordingly.

Notably, in one embodiment as mentioned above, body tracking algorithms,such as skeletal tracking algorithms, may be utilized to track aperson's depth as the person moves around the field of view of thecameras. For example, in one embodiment, the perspective (relative size)of the skeletally tracked individual(s) (once focused on that particularindividual within the desired depth range) may result in correspondingchanges to the depth range: for instance, a decrease in size impliesmovement away from the camera, and thus a corresponding increase infocus depth, while an increase in size implies movement toward thecamera, and thus a corresponding decrease in focus depth. Other skeletaltechniques may also be used, such as simply increasing or decreasing thedepth (e.g., scanning the focus depth toward or away from the camera) orby increasing the overall size of the depth range (e.g., moving one orboth of the near and far depth thresholds in a manner that widens thedepth range).

In an alternative embodiment, if body tracking is enabled, the set depthrange may remain the same, but a person's body that leaves that depthrange may still be tracked, and isolated from the remaining sceneoutside of the depth range. For instance, body tracking algorithms maybe used to ensure a person remains “captured” even if they step out ofthe specified depth range, allowing for certain objects to be left inthe depth range for capture while a person has the freedom to move outof the depth range and still be captured. As an example, assume in FIG.13C that there was an object, such as a chair, within the specifieddepth range 1330. If the person were to step out of the depth range 1330while body tracking in this embodiment was enabled, the chair wouldremain in the isolated portion of the scene, as well as the person'sbody, regardless of where he or she moved within the captured imagespace. On the contrary, in the embodiment above where the body trackingadjusts the depth range, the chair may come into “view” of thedynamically adjusted depth range 1330 and become part of the isolatedimage only when the person moves to a depth corresponding to the chair.

Accordingly, with either type of body tracking enabled, an operatorwould not need to manually adjust the min/max depth to retain performersin a scene. For example, once the depth range is set, if body trackingis enabled and a person moves out of the depth range, they will still betracked and included within the cut-out footage, whether by dynamicallyadjusting the depth range, or else by specifically following theperson's body throughout the captured scene. (Note that the manual depthadjustments or “sliders” to set the near and far thresholds may remainavailable for including non-body objects in the scene.)

In accordance with one or more additional embodiments described herein,other filtering features may further adjust the area of the resultantimage 1340, such as by managing a Gaussian function, a “disc blur”effect, or other techniques to smooth and/or sharpen the edges of thearea isolated from the video image 1310. Other advanced techniques arealso possible, such as skeletal tracking algorithms, which will enable abetter picture and closer cutout of an individual in the desired depthrange. By adding the ability to soften and blur the edges of the cut-outimages, displaying (or overlaying) the depth-isolated image has edgesthat look smooth/realistic.

Additional image processing features are also made available by thetechniques herein, in order to provide greater functionality. Forinstance, in one embodiment, the video camera 1210 and IR camera 1224(e.g., and optionally IR emitter 1222 or else the entire depth capturingcomponent 1220) may be rotated vertically to achieve greater resolutionwhen filming a standing person (e.g., such that the aspect ratio of thecameras is oriented in a vertically extended manner), for example, whenobjects to either side of the standing person are not required.Accordingly, in this embodiment, the final cut-out image may be rotated(e.g., 90 degrees) so the person/object is in the correct orientationwhen projected/overlayed in its final display application (e.g.,described below). In addition, in another embodiment, the cut-out imagecan be flipped (e.g., horizontally and/or vertically) to displaycorrectly (for example, when filming a guitarist, the displayed imagemay need to be flipped to show the guitarist playing on the correcthanded guitar, depending upon the method of display, e.g., projection,reflection, digital processing, etc.). Still further, in one embodiment,the cut-out image may be resized to make the person/object a realisticsize when it's displayed (e.g., bigger or smaller, wider or thinner,taller or shorter). Moreover, in yet another embodiment, post-processingtechniques may be used to add scenes around the cut-out image, such asmaking the final result a “full-screen” image (e.g., a cut-out personstanding in a generated or separately filmed background scene, etc.).For instance, in one specific example, a “floor” may be input beneath aperson/object and shadows may be added on the floor (e.g., moving orstationary) to create a more realistic visual effect (particularly forholographic images), such as what is shown in FIG. 14.

With general reference to the techniques described above, FIG. 15illustrates an example simplified procedure for depth key compositing inaccordance with one or more embodiments described herein. The procedure1500 may start at step 1505, and continues to step 1510, where, asdescribed in greater detail above, a digital visual image is capturedfrom a video capture device. Illustratively, in one embodiment, in step1515 a capture space of the captured digital visual image may be bathedwith infrared (IR) light from a source located at the video capturedevice (e.g., integrated with the video capture device), and in step1520 a brightness of IR light reflected off of objects within thecapture space in order to define the particular depth range as acorresponding range of reflected IR brightness in step 1525 (e.g.,manually adjusting with distance thresholds and/or dynamically adjustingwith object tracking algorithms).

In step 1530, one or more objects within the digital visual image aredetermined that are within a particular depth range of the video capturedevice. In one specific embodiment, determining the one or more objectswithin the digital visual image that are within the particular depthrange of the video capture device is based on the one or more objectshaving a particular reflected IR brightness within the correspondingrange of reflected IR brightness of the particular depth range.

In step 1535, the one or more objects may be isolated from portions ofthe digital visual image not within the particular depth range, and theone or more isolated objects may be processed in step 1540 for visualdisplay apart from the portions of the digital visual image not withinthe particular depth range. For example, as noted above, such processingmay comprise applying image filtering, rotating, flipping, re-sizing,adding other images around the one or more isolated objects, preparingthe one or more isolated objects for holographic displays, and so on.

The simplified procedure 1500 ends in step 1545, notably with the optionto continue to capture images, isolate objects, track objects, adjustdepth ranges, etc. Also, the processing in step 1540 may continue, suchas storing the isolated (and processed) images, displaying the isolatedimages, streaming the isolated images, and so on, such as for filmproduction and/or holographic displays.

It should be noted that while certain steps within procedure 1500 may beoptional as described above, the steps shown in FIG. 15 are merelyexamples for illustration, and certain other steps may be included orexcluded as desired. Further, while a particular order of the steps isshown, this ordering is merely illustrative, and any suitablearrangement of the steps may be utilized without departing from thescope of the embodiments herein.

Returning specifically to the pyramid holographic projection embodimentsherein, the “hologram selfie” technology above (removing the user orother object from its background for display as a clean hologram) may bespecifically paired with the pyramid hologram production techniquesabove. In particular, and with reference to the procedure 1600 outlinedin FIG. 16, the selfie creation process (e.g., depth-keying videocapture of a subject) may capture one or more (e.g., 1, 2, or 4) videosat a time in step 1610. Depending on the purpose of the video, theprocedure may either stream the selfie video in step 1615, or else mayrecord/store the video in step 1620. Visual effects (VFX) and/or specialeffects (SFX) may optionally be injected in step 1625 prior toconverting the selfie video into a single selfie pyramid video in step1630. In particular, as described in detail above, step 1630 maycomprise duplicating the video to the number needed for the pyramid(e.g., ensuring four copies for a four-sided pyramid) in sub-step 1632,and also scaling, cropping, and rotating the video images, andoverlaying them on top of an empty background to form an open box layout(e.g., also adjusted for aspect ratio, resolution, etc., at runtime) insub-step 1634. Then, in steps 1635 or 1640, the pyramid selfie video maybe streamed or stored (e.g., on the cloud or otherwise), respectively.

For example, as shown in FIG. 17, the techniques herein may be used forpersonal communication (e.g., selfie holographic video chat,telepresence, etc.), whether real-time/live or pre-prerecorded (andstored locally with the case, or accessed from a networked database).For example, in a communication system 1700, a source user 260 a mayhave a corresponding source camera device 1710, such as a smartphone,tablet, specifically designed device (e.g., with depth sensingtechnology), free-standing camera, and so on. The source images may becaptured, and communicated through a network 530 to the destinationdisplay device 1720 (e.g., tablet, phone, etc.). By processing thecaptured image (e.g., on the source device, within the network, or atthe destination device), the resultant pyramid image may be displayed onthe pyramid holographic projection device 1730 to one or more receivingusers 260 b as shown. Note that in the case of stored images/video(e.g., selfie messages, avatar recordings, stored character animations,etc.) database 1740 may receive, store, and later transmit the desiredimages/video from the source camera 1710, accordingly.

Note that in one specific embodiment, a travel case for a portablepyramid holographic projection setup may be used as described hereinthat provides a road case that folds out into a pyramid holographicprojection system, allowing for extended portability of larger (largerthan smartphone or tablet) communication system endpoints. Specifically,in one embodiment, the portable case may be built for air travel, whichcurrently must meet the weight and dimension restrictions of being lessthan 50 pounds and less than or equal to 62 linear inches total inheight, width, and length (H+W+L) (i.e., the current baggage restrictionfor normal checked luggage for many major airlines). The setup herein,in particular, illustratively uses a video panel display and definesspace-saving designs for legs, a folding or assemble-able holographicpyramid and frame, as well as for other components (e.g., remotes,wires, etc.).

For example, with reference to FIG. 18, a travel case 1800 is shown,particularly where the outside 1810 of the travel case is shown fromvarious angles, and generally may comprise a handle 1820, variouslatches 1830, and so on. In one embodiment, the outside of the travelcase 1810 may be built in a manner similar to road cases for musiciansand/or audio/video equipment, as will be appreciated by those skilled inthe art, e.g., where the corners are reinforced with metal, the latchesare recessed into the case, etc.

As shown in FIGS. 19A-19B, when opened, the travel case 1800 providesfor a portable pyramid holographic projection setup. In particular, avideo panel display (e.g., LED, LCD, etc.) may be mounted to the tophalf (“lid”) of the case 1810/1910, and provides the image source forthe pyramid holographic image 1935 that is reflected off the pyramidholographic projection screen/surfaces 1930. In addition, the lid of thecase 1810/1910 may have a supporting bracket that holds the pyramidholographic projection surfaces 1930 (e.g., separate sheets of plastic,glass, etc. converging to a support, not shown), each on theillustrative 45-degree angle to the image source (extending angularlyaway from the image source). (Note that a base 1810/1915 or leg assembly1920 may be provided for the bottom of the holographic screen 1930 aswell for greater support, though the pyramid may rest on the floor ormay free-hang from the lid.) The bottom half (“base”) of the case1810/1915 may be used as a stage or floor for the system, or else may besimply placed aside until the system is repacked. Alternatively, foampacking material (as described below) may also be used as a stage forthe system. Note that though the travel case is shown with the displayon the top/lid, a reverse configuration may also be used herein (i.e.,the display on the bottom/based, with the pyramid facing the otherdirection, accordingly).

Various alternative applications of the techniques herein are alsospecifically contemplated herein. For example, the pyramid holographicprojection herein may be applied to a variety of environments, whetherfor film production, live streaming, simulcasts, or pre-recordedapplications. For instance, with reference again to FIG. 5, a broadcastvenue may comprise the source A/V components 510, such as where aperformance artist is performing (e.g., where a disc jockey (DJ) isspinning) in person. The techniques herein may then be used to stream(relay, transmit, re-broadcast, etc.) the audio and video from thisbroadcast location to a satellite venue, where the remote A/V components550 are located. For instance, the DJ in the broadcast location may havethe associated audio, video, and even corresponding electronic effects(lights, pyrotechnics, etc.) streamed directly to the satellite venue'sA/V system with the same high quality sound as if the musician/artistwas playing/singing in person.

By streaming the video image of the performer as a video and projectingit onto a holographic projection system, a true concert or nightclubexperience can be transmitted across the globe for the liveentertainment experience. For example, holographically live-streamingconcerts to satellite venues around the globe while maintaining the liveconcert experience helps artists reach new markets and new revenuestreams, while bringing live sets to more fans all across the world.Satellite venues can be configured to have the same concert feel as anactual show: intense lighting effects, great sound quality, bars,merchandise, etc. The only difference is that the performers are notphysically present, but are holographically projected from the broadcastvenue. The music is streamed directly from the soundboard of thebroadcast venue and sent to state-of-the-art sound systems at thesatellite venues. Light shows may accompany the performance with top ofthe line LED screens and lasers.

For example, once the desired image is obtained from the techniquesabove, the desired image may be imported into an encoding software thatallows for live streaming of video, while the accompanying audio may bebrought into the computer and program separately. In one embodiment, thevideo/audio transmission may be directly to the remote/satellitecomputer, or else may be uploaded to a secure webpage first, and thendownloaded from the remote site(s), such as by opening this webpage on asecure computer at the satellite venues.

In addition to concerts and nightclubs, the techniques herein may alsobe used for retail spaces, movie special effects, tradeshows, movietheater lobbies, conferences, speeches, retail window displays, personalappearances, and so on.

According to one or more embodiments described herein, therefore, thetechniques herein provide for real-time video processing for pyramidholographic projections by:

-   -   obtaining one or more rectangular input images;    -   determining a geometry of a pyramid holographic projector;    -   cropping, by the process, the one or more rectangular input        images into one or more corresponding trapezoidal input images        with a flat planar UV map based on the geometry of the pyramid        holographic projector; and    -   producing, by the process, a pyramid holographic projection        image source for projection on the pyramid holographic        projector, the pyramid holographic projection image source        stitching the one or more corresponding trapezoidal input images        together on top of a transparent frame, wherein each of the one        or more corresponding trapezoidal input images are positioned        and rotated to form a shape corresponding to the geometry of the        pyramid holographic projector, wherein the shape is centered in        the transparent frame.

In addition, in certain embodiments, obtaining comprises one or more ofreal-time image capture or stored image retrieval.

In addition, in certain embodiments, the process further comprisescopying one or more of the trapezoidal input images to be stitched andprojected on more than one side of the pyramid holographic projector. Inaddition, in certain embodiments, a same trapezoidal input image isstitched and projected on all sides of the pyramid holographicprojector.

In addition, in certain embodiments, obtaining comprises capturing avideo selfie of a user. In addition, in certain embodiments, the processcomprises separating an image of the user from a background environment(e.g., based on depth-keying or other separation technique selected froma group consisting of: chroma-keying; skeletal recognition; and the userbeing in front of a black background).

In addition, in certain embodiments, the one or more rectangular inputimages are selected from a group consisting of: a single image; aplurality of separate images; a video stream of images; and a pluralityof video streams of images.

In addition, in certain embodiments, the one or more rectangular imagesare selected from a group consisting of: images of a user; images of anobject; images of an avatar; and images of an animated character.

In addition, in certain embodiments, the process may further compriseperforming one or more image processing techniques on one or both of theone or more rectangular input images or one or more correspondingtrapezoidal input images.

In addition, in certain embodiments, one or more of determining,cropping, and producing are performed in real-time with the obtaining.

In addition, in certain embodiments, one or more of determining,cropping, and producing are performed during post-processing afterobtaining and storing the one or more rectangular input images.

In addition, in certain embodiments, the process may comprise streamingthe produced pyramid holographic projection image source for real-timeprojection on the pyramid holographic projector.

In addition, in certain embodiments, the process may comprise storingthe produced pyramid holographic projection image source for playbackprojection on the pyramid holographic projector.

In addition, in certain embodiments, the geometry of a pyramidholographic projector is selected from a group consisting of: athree-sided pyramid; a four-sided pyramid; a three-sided pyramid withone side being a wall without a projected image; and a four-sidedpyramid with one side being a wall without a projected image.

Advantageously, the techniques herein provide for real-time videoprocessing for pyramid holographic projections for various applications,such as film, live streaming entertainment systems, and so on. Inparticular, as described above, the techniques herein provide theability to create a real holographic selfie that can be streamed orplayed-back on any smartphone, tablet, or even bigger screens. In thismanner, the techniques herein can be implemented for real-timeholographic calls, holographic messaging, telepresence, and so on, wherevideo images of a user are captured, converted to a specific videoformat, and played as a pyramid hologram.

In addition, for performance artists, live streaming an event tosatellite locations, particularly holographically, is a great way toincrease exposure while gaining an additional revenue stream withoutadded cost. Moreover, receiving a holographic live stream at a venue maybe at a fraction of the cost of paying the performance artist(s) toappear in person. Moreover, the ability to draw attention to displayedimages, such as for marketing or consumer experience (e.g., sportingevents, concierge services, shopping displays, etc.), is a key goal formany industries.

While there have been shown and described illustrative embodiments, itis to be understood that various other adaptations and modifications maybe made within the spirit and scope of the embodiments herein. Forexample, the embodiments described herein may be used with holographicprojection images produced from a variety of sources, such aslive-streamed, pre-recorded, re-constructed, computer-generated, and soon. Also, any reference to “video” or “image” or “picture” need notlimit the embodiments to whether they are motion or time-sequencephotography or still images, and so on. (That is, while the embodimentshave been generally described in terms of video capture, still pictures(stationary images) may also benefit from the techniques herein.)Furthermore, any multi-faceted holographic imagery device may be usedherein, and the illustrations provided above are merely exampleembodiments, whether for four-sided pyramid objects (and correspondingimages) or otherwise.

Moreover, the embodiments herein may generally be performed inconnection with one or more computing devices (e.g., personal computers,laptops, servers, specifically configured computers, cloud-basedcomputing devices, cameras, etc.), which may be interconnected viavarious local and/or network connections. Various actions describedherein may be related specifically to one or more of the devices, thoughany reference to particular type of device herein is not meant to limitthe scope of the embodiments herein.

For example, FIG. 20 is a schematic block diagram of an examplecomputing device 2000 that may be used with one or more embodimentsdescribed herein. The illustrative device may comprise at least onenetwork interface 2010, one or more audio/video (A/V) interfaces 2015,at least one processor 2020, a memory 2030, and user-interfacecomponents 2070 (e.g., keyboard, monitor, mouse, etc.), interconnectedby a system bus 2080, as well as a power supply 2090. Other componentsmay be added to the embodiments herein, and the components listed hereinare merely illustrative.

The network interface(s) 2010 contain the mechanical, electrical, andsignaling circuitry for communicating data over links coupled to acomputer network. A/V interfaces 2015 contain the mechanical,electrical, and signaling circuitry for communicating data to/from oneor more A/V devices, such as cameras, soundboards, lighting boards,display projectors, etc. The memory 2030 comprises a plurality ofstorage locations that are addressable by the processor 2020 for storingsoftware programs and data structures associated with the embodimentsdescribed herein. The processor 2020 may comprise hardware elements orhardware logic adapted to execute the software programs and manipulatethe data structures 2039. An operating system 2032, portions of whichare typically resident in memory 2030 and executed by the processor,functionally organizes the machine by invoking operations in support ofsoftware processes and/or services executing on the machine. Thesesoftware processes and/or services may comprise an illustrative pyramidholographic projection process 2034, a real-time streaming process 2036,and A/V processing process(es) 2038.

It will be apparent to those skilled in the art that other processor andmemory types, including various computer-readable media, may be used tostore and execute program instructions pertaining to the techniquesdescribed herein. Also, while the description illustrates variousprocesses, it is expressly contemplated that various processes may beembodied as modules configured to operate in accordance with thetechniques herein (e.g., according to the functionality of a similarprocess). Further, while the processes have been shown separately, thoseskilled in the art will appreciate that processes may be routines ormodules within other processes.

Illustratively, certain aspects of the techniques described herein maybe performed by hardware, software, and/or firmware, such as inaccordance with the various processes and components described herein,which may contain computer executable instructions executed by theprocessor 2020 and/or associated hardware components to performfunctions relating to the techniques described herein.

The foregoing description has been directed to specific embodiments. Itwill be apparent, however, that other variations and modifications maybe made to the described embodiments, with the attainment of some or allof their advantages. For instance, it is expressly contemplated thatcertain components and/or elements described herein can be implementedas software being stored on a tangible (non-transitory)computer-readable medium (e.g., disks/CDs/RAM/EEPROM/etc.) havingprogram instructions executing on a computer, hardware, firmware, or acombination thereof. Accordingly this description is to be taken only byway of example and not to otherwise limit the scope of the embodimentsherein. Therefore, it is the object of the appended claims to cover allsuch variations and modifications as come within the true spirit andscope of the embodiments herein.

What is claimed is:
 1. A method, comprising: obtaining, by a process,one or more rectangular input images; determining, by the process, ageometry of a pyramid holographic projector; cropping, by the process,the one or more rectangular input images into one or more correspondingtrapezoidal input images with a flat planar UV map based on the geometryof the pyramid holographic projector; and producing, by the process, apyramid holographic projection image source for projection on thepyramid holographic projector, the pyramid holographic projection imagesource stitching the one or more corresponding trapezoidal input imagestogether on top of a transparent frame, wherein each of the one or morecorresponding trapezoidal input images are positioned and rotated toform a shape corresponding to the geometry of the pyramid holographicprojector, wherein the shape is centered in the transparent frame. 2.The method as in claim 1, wherein obtaining comprises one or more ofreal-time image capture or stored image retrieval.
 3. The method as inclaim 1, further comprising: copying one or more of the trapezoidalinput images to be stitched and projected on more than one side of thepyramid holographic projector.
 4. The method as in claim 3, wherein asame trapezoidal input image is stitched and projected on all sides ofthe pyramid holographic projector.
 5. The method as in claim 1, whereinobtaining comprises: capturing a video selfie of a user.
 6. The methodas in claim 5, further comprising: separating an image of the user froma background environment.
 7. The method as in claim 6, whereinseparating is based on depth-keying.
 8. The method as in claim 6,wherein separating is based on a separation technique selected from agroup consisting of: chroma-keying; skeletal recognition; and the userbeing in front of a black background.
 9. The method as in claim 1,wherein the one or more rectangular input images are selected from agroup consisting of: a single image; a plurality of separate images; avideo stream of images; and a plurality of video streams of images. 10.The method as in claim 1, wherein the one or more rectangular images areselected from a group consisting of: images of a user; images of anobject; images of an avatar; and images of an animated character. 11.The method as in claim 1, further comprising: performing one or moreimage processing techniques on one or both of the one or morerectangular input images or one or more corresponding trapezoidal inputimages.
 12. The method as in claim 1, wherein one or more ofdetermining, cropping, and producing are performed in real-time with theobtaining.
 13. The method as in claim 1, wherein one or more ofdetermining, cropping, and producing are performed duringpost-processing after obtaining and storing the one or more rectangularinput images.
 14. The method as in claim 1, further comprising:streaming the produced pyramid holographic projection image source forreal-time projection on the pyramid holographic projector.
 15. Themethod as in claim 1, further comprising: storing the produced pyramidholographic projection image source for playback projection on thepyramid holographic projector.
 16. The method as in claim 1, wherein thegeometry of a pyramid holographic projector is selected from a groupconsisting of: a three-sided pyramid; a four-sided pyramid; athree-sided pyramid with one side being a wall without a projectedimage; and a four-sided pyramid with one side being a wall without aprojected image.
 17. A tangible, non-transitory computer-readable mediacomprising instructions executable by a processor for executing aprocess comprising: obtaining one or more rectangular input images;determining a geometry of a pyramid holographic projector; cropping theone or more rectangular input images into one or more correspondingtrapezoidal input images with a flat planar UV map based on the geometryof the pyramid holographic projector; and producing a pyramidholographic projection image source for projection on the pyramidholographic projector, the pyramid holographic projection image sourcestitching the one or more corresponding trapezoidal input imagestogether on top of a transparent frame, wherein each of the one or morecorresponding trapezoidal input images are positioned and rotated toform a shape corresponding to the geometry of the pyramid holographicprojector, wherein the shape is centered in the transparent frame. 18.The computer-readable medium as in claim 17, wherein obtaining comprisesone or more of real-time image capture or stored image retrieval, andwherein the process further comprises one of either: streaming theproduced pyramid holographic projection image source for real-timeprojection on the pyramid holographic projector; or storing the producedpyramid holographic projection image source for playback projection onthe pyramid holographic projector.
 19. The computer-readable medium asin claim 17, wherein obtaining comprises: capturing a video selfie of auser; and separating an image of the user from a background environment.20. The computer-readable medium as in claim 17, wherein one or more ofdetermining, cropping, and producing are performed either in real-timewith the obtaining or during post-processing after obtaining and storingthe one or more rectangular input images.